Guideline: In More Details - Determine The Planning (AST)

In creating a planning, the following principles apply:

The test strategy and estimated effort form the basis of the setup of the planning
In a good planning, the characteristics/object parts designated as high risk are tested as early as possible
With optimal planning, as far as possible only the test execution activities are carried out on the critical path of the project
If there are no past figures available in respect of the number of retests, it is advisable to make allowance when creating a planning with one retest on average. Based on past experience, it can be decided to allow more or less time for retesting.
With maintenance, in particular, and with iterative or agile development phases, it is important to make allowance for the execution of regression tests
When creating a planning, make allowance for the required time of third parties. For example, repair time for defects or time for preparing the test environment
The transfer of the test object to, and installation in, the test environment often falls between two stools in planning, or rather between the planning of the development and testing activities. Particularly the first few times, this activity appears to cost significant amounts of time – days rather than hours. Make allowance for this.
Try to streamline the in- and outflow of personnel, so that peaks and troughs in staff levels are avoided.
Further planning indications can be found in the IT classic [Brooks, 1975/1995].

Required information

In order to set up a planning based on the estimated effort, additional information is required concerning the following subjects:

Available resources - Worth noting here is that with the estimation of the effort, only limited allowance is made for the available resources. The calculation is made on the number of hours required. In combination with a deadline, this means that a certain number of resources is required for carrying out the planned tests. In practice, it is often the case that the number of available resources initially does not correspond with the required amount of resources. The test manager should make this clear and then discuss it with the client. Possible solutions are the hiring of temporary personnel, extending the timeline or adjusting the strategy.
Available timeline - In practice, the available timeline is usually provided in the form of a deadline for the relevant phase.
Availability of resources, such as test environments and test tools - When are these to be available for the activities? Do the test tools, for example, still have to be selected, purchased and set up?
Dependencies between the various activities - Activities that depend on other activities can only start after completion of those other activities and not in parallel with them.
Method of system development - The test levels are planned depending on the way in which the system is developed. With a waterfall method, the phasing is different from that of an iterative process in which testing and development activities are parallel and sometimes executed integrally. The development test and system test as a rule have more to do with this than the acceptance test.
Information on milestones in the development project - This information is necessary in order to coordinate the test planning optimally with the planning of the rest of the project. This makes it possible to minimise the total timeline of the project.

Exit criteria can relate, for example, to the number of issues in a particular risk category that may still be open, the way in which a certain risk is covered (e.g. all the system parts with the highest risk category have been tested using a formal test design technique), or the depth in which the requirements should have been tested. From within the master test plan, the exit criteria are applied to the test level. If that is not the case, or if there is no master test plan, the test manager should agree the criteria with the client.

The box below shows a number of concrete examples of exit criteria:

System X may only be transferred to the AT when the following conditions have been met:

There are no more open defects in the category of “severe”
There is a maximum of 4 open defects in the category “disrupting”
The total number of open defects is no more than 20
A workaround has been described for every open defect
For every user functionality, at minimum, the correct paths have been tested and approved

System X may be transferred to the AT when it can be shown in writing that all the risks that were allocated to the ST in accordance with document Y have been tested in the agreed depth and by the agreed test method.

An important point of focus as regards the above-mentioned criteria is that clear defi nitions should be agreed by all the stakeholders of what a particular category of severity is and what is meant by ‘agreed depth of testing and test method’. In practice, a lack of clarity here can lead to heated discussions.

Similarities and differences between acceptance and exit criteria

Another term for exit criteria that is used is ‘acceptance criteria’, as discussed in Understand The Assignment (AST). Besides the fact that acceptance criteria may be a broader term than exit criteria, another difference is that acceptance criteria come at the end, i.e. at acceptance, and exit criteria at the transfer from one test level to another, or to production. The figure below illustrates this.

Figure 1 - Exit and acceptance criteria